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(54) Method and system for maintaining freeform ink annotations on changing views 



(57) The method and system ol the invention main- 
tains the indication of the logical relationship of freeform 
digital ink annotations to objects in an electronic display 
even as the view changes. The invention maps the log- 
ical relationship of the freeform digital ink annotations 
to objects in the view based upon the layout relation- 



ships of the freeform digital ink annotations to the ob- 
jects in the view. If the view changes, the invention main- 
tains the layout relationship of the freeform digital ink 
annotations to the objects in the view based upon the 
mapping of the freeform digital ink annotations to the 
objects. 
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Description 

[0001] This application is directed to an electronic system lor displaying changing views. In particular, the invention 
is directed to permitling Ireeform digital ink annotalion on electronic media and to maintaining the logical relationship 
between freeform digital ink annotations and objects in changing views of the media. 

[0002] Freeform ink annotalion ol paper documents is a wide-spread practice because it aids many aspects of doc- 
ument work, including the processes of thinking, categorizing and reviewing. In short, freeform ink annotation has 
proven to be a powerful comprehension tool. Paper and paper-like documents (such as PDF, postscript, image files 
etc.) have a static layout and content. The static nature of these documents makes annotation simple. 
[0003] However, the kinds of documents that people use are increasingly created and presented digitally on displays 
using computers. Such documents are often dynamic and. therefore, the accurate representation of the relationship 
of the annotations to objects in the display has been impossible because of the changing display For example, a 
display of a word processor document changes as the document is undergoing revisions. In a similar mannei. infor- 
mation visualization systems such as the hyperbolic browser disclosed in A Focus + Context Technique Based on 
Hyperbolic Geometry for Visualizing Large Hierarchies, by Lamping et aL, in Proceedings of ACM SIGCHI Conference 
on Human Factors and Computing Systems, May 1995, change the display of the document as the user navigates the 
information space. 

[0004] A freeform digital ink whiteboard known as Tivoli is disclosed in Pen- Based Interaction Techniques For Or- 
ganizing Material on an Electronic Whiteboard, by Moran, et aL. in UIST "97. ACM press. Trvoli allows users to identify 
a connection between two areas in a display, where that connection is maintained as those areas are moved and the 
display changes. Circle marks which surround other ink identify areas, and freeform digital ink marks which connect 
two area boundaries together form a connection between those two areas. This connection persists as the areas move 
and Tivoli attempts to maintain the shape of the connecting mark. 

[0005] To assist in navigation of documents conventional systems use a set type of bookmark. Examples of such 
navigation systems are disclosed in Virtual Annotation System, by Harmon, et aL, in Proceedings of the IEEE 1996 
Virtual Reality Annual International Symposium, pp. 239-245 (1996) and in Worldlets:3D Thumbnails for Wavfindino 
in Virtual Environments, by Elvins. et aL, in video proceedings ol UIST '97, pp. 21-30 (1997). These systems make it 
ackward for a user to freely interact with the navigation system because the user is required to specify an annotation 
command before commenting. 

[0006] Current systems attempt to manage freeform digital ink annotation on changing views or displays by at least 
one of three methods (the terms "view - and 'display" are used interchangeably throughout this application). The first 
method, known as the static approach, treats ink annotations as associated with a single view When the presentation 
or the content of the document changes, the annotations either disappear or remain in their original positions. A system 
known as "MATE* which is disclosed in A Marking Based Interface for Collaborative Writing , by Hardock et al , in 
Proceedings of UIST '93 (1993), addresses changing content by maintaining a static annotated version, a changed 
version without annotation and logical links between these two versions. 

[0007] A second method, known as the snapshot approach, treats the dynamic view as a series of static snapshots 
which can be individually annotated. Examples of this approach include the Wang Freestyle disclosed in Rapid, Inte- 
grated Design of a Multimedia Communication System, by Francik, in Human-Computer Interface Desiqn Morgan 
Kaufman Publishers, Inc., (1 996). ' 
[0008] The third method, known as the temporal association approach, separates the ink annotations from the view 
and links the two temporally, so that marks made in a specific time are associated with the view at that time. Examples 
of this approach are used in Marquee which is disclosed in Marguee: A Tool for Real-Time Video Logging by Weber 
et al., in Proceedings of CHI '94, ACM Press. (April 1 994) and Dynomite which is disclosed in Dynomite- A Dynamically 
Organized Ink and Audio Notebook, by Schilii et aL, In Proceedings of CHI '97, ACM Press. (1997). 
[0009] In conventional interlaces thai accept freeform digital ink annotation, the indication of the logical relationships 
between the annotations and the associated objects are not maintained il the view changes. For example, if a user 
annotates the text of a document and, subsequently, the annotated text reflows across lines, the annotation does not 
follow the reflowed portion of the text. Therefore, while there may be an underlying logical relationship to the annotated 
text, the layout of the new view does not indicate this logical relationship between the text and the annotation. 
[0010] The systems described as related art above also do not maintain any indication of logical relationships be- 
tween the annotations and the associated objects if the view changes. Tivoli does not maintain the freeform digital ink 
annotations as the areas move around and the display changes. Rather. Tivoli maintains only one specific type of 
annotation: a single stroke from the boundary of one area to another. This restriction precludes the use of freeform 
digital ink annotations in changing displays to label moving objects or to aid navigation. 

[0011] While MATE maintains the logical relationships between freeform digital ink annotations and the corresponding 

underlying text, it does not maintain the indications of these logical relationships as the view changes. 

[0012] Freestyle does not allow the freeform digital ink annotations to carry over to new views of the same document 
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[001 3] Dynomite does not permit annotations directly on the view and the annotations are not rendered on the new 
views in accordance with changes to the views. 

[0014] The method and system of the invention maintain the layout relationships between freeform digital ink anno- 
tations and objects in changing views. As a user makes a freeform digital ink annotation on a view and as the view 
changes, the method and system of the invention adapt the marks in accordance with the changes. Thus, the logical 
relationship between the freeform digital ink annotations and the objects made in one view can be understood in future 
views based upon the layout relationships. 

[0015] The method and system also maintain the indication of the logical relationships between freeform digital ink 
annotatons and objects in dynamic media such as video and interactive virtual worlds. For example, if a user, who is 
exploring an information space with a hyperbolic browser, makes a freeform digital ink annotation on a node of the 
tree, when the user changes the display to bring another portion of the tree into focus, the freeform digital ink annotation 
rotates and scales along with the tree to continue to indicate the logical relationship of the freeform digital ink annotation 
to the node. 

[001 6] The methods and systems of this invention maintain the indication ol the logical relationships between freeform 
digital ink annotations and objects in a view as the view of the annotated objects changes. This invention brings the 
experience of marking on paper to dynamic media, changing word-processor documents, information visualizations 
virtual worlds, digital video and word processors. 

[001 7] The system and method of the present invention maintain the indication of the logical relationship, even if the 
objects that are logically related to the freeform digital ink annotations are deleted, inserted, copied or moved or oth- 
erwise altered. The method and system also maintain the indication of the logical relatbnshp if the layout presentation 
of the objects in the display change because the objects reflow, scale, resize, rotate, stretch, distort or morph or if any 
other aspect of the layout presentation changes, regardless of whether these changes occur discretely or continuously. 
[0018] Additionally, another exemplary embodiment of the systems and methods of the invention provides naviga- 
tional markers that make freeform digital ink annotation a powerful tool for navigating dynamic documents. A user is 
permitted to select a freeform digital ink annotation in an arbitrary view, and the display is returned to the state it was 
in when the selected freeform digital ink annotation was made. 

[001 9] The preferred embodiments of this invention will be described in detail, with reference to the following figures 
wherein: 

Fig. 1 is a block diagram of an embodiment of the system for displaying changing views of this invention; 
Fig. 2A shows a display of an annotated hyperbolic browser in accordance with an embodiment ol the present 
invention; 

Fig. 2B shows the annotated hyperbolic browser of Fig. 2A with the view changed and freeform digital ink anno- 
tations adjusted in accordance with the change in the view in accordance with an embodiment of the invention- 
's Fig. 3A shows a display of an annotated document; 

Fig. 3B shows the annotated document of Fig. 3A after modification of the document and shows the adjustments 
to the freeform digital ink annotations in accordance with an embodiment of the present invention; 
Figs. 4A-4F show a flowchart outlining a control routine for mapping freeform digital ink annotations to objects in 
a display in accordance with an embodiment of the prevent invention; 

Figs. 5A-5B show a flowchart outlining a control routine for rendering freeform digital ink annotations on a changed 
view in accordance with an embodiment of the present invention; and 

Fig 6 shows a few examples ol different types of freeform digital ink annotations. These and other features and 
advantages of this invention are described in or are apparent from the following detailed description of the preferred 
embodiments. 

[0020] Fig. 1 shows one exemplary embodiment of the electronic system 1 0 of this invention. The electronic system 
10 includes a processor 12 communicating with a memory 14 that stores the programs, documents and other data for 
practicing the invention. The processor 12 also communicates with a display 16. a keyboard 18, a mouse 20 and a 
pen 22 via input/output device 24. The keyboard 18, mouse 20. pen 22 and any other interface devices (not shown) 
are operated by a user to control the operation of the electronic system 10. Control of the electronic system 10 is not 
limited to the interface devices shown and does not require any particular interface device. The display 16 may be on 
a stand-atone pen computer or a tablet connected by a tether to a conventional computer or any other type of display. 
[0021] As shown in Fig. I.thesystem 1 0 is preferably implemented using a prograrrimed gerieral purpose computer. 
However, the system 10 can also be implemented using a special purpose computer, a programmed microprocessor 
or mjcrocontroner and any necessary peripheral integrated circuit elements, an ASIC or other integrated circuit, a 
hardwired electronic or logic circuit such as a discrete element circuit, a programmable logic device such as a PL0. 
PLA FPGA or PAL, or the like. In general, any device on which a finite state machine capable of implementing the 
flow charts shown in Figs. 4A-5B can be used to implement the system 10. 
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[0022] Additionally, as shown in Fig. 1, the memory 14 is preferably implemented using static or dynamic RAM. 
However, the memory 14 can also be implemented using a floppy disk and disk drive, a writable optical disk and disk 
drive, a hard drive, a flash memory or the like. Additionally, it should be appreciated that the memory 14 can be either 
distinct portions of a single memory or physically distinct memories. 

[0023] Further, it should be appreciated that the links 26. 28. 30, 32. 34 and 36 connecting the memory 1 4. the display 
16, the keyboard 18, the mouse 20. the pen 22 and the input/output device 24 to the processor 12 can be wired or 
wireless links to networks (not shown). These networks can be local area networks, wide area networks, intranets, the 
I nternet, or any other distrbuted processing and storage networks. 

[0024] Fig. 2A shows a display of a view ol a hyperbolic browser 40. The display 40 includes a f reeform digital ink 
annotation 42. Fig. 2B shows an alternative view of the document shown in Fig. 2A. The f reeform digital ink annotation 
42 has been scaled and moved so that the layout presentation of the freefonm digital ink annotation in the view corre- 
sponds with the change of the view of the objects in the document 40 to maintain the indication of the logical relationship 
between the freelorm digital ink annotation 42 and the objects. 

[0025] Fig. 3A shows a first view 50 of a document 52 having freeform digital ink annotations 54, 56 and 58 Fig 3B 
shows a second view 60 of the document 52 after the document has been revised. Fig. 3B shows that the freeform 
digital ink annotations 54, 56 and 58 are adjusted in accordance with the logical relationships to the objects in the 
document 52 in accordance with an embodiment o1 the invention. The content of the document 52 shown in Fig. 3A 
has been edited to insert the text 'is so much fun to do" at 62 and to delete approximately two full lines of text at 64. 
In response to these edits, the text that has been annotated at 54 now flows across two lines and the •cross-ouf 
freeform digital ink annotation 54 flows along with the underlying text objects to continue to indicate the logical rela- 
tionship between -cross-out' freeform digital ink annotation and the underlying text objects. The invention also recog- 
nizes that the group of ink strokes at 58 is a margin note and the system slightly adjusts the vertical position of the 
margin note58 in the margin to continue to indicate the logical relationship between the margin note 58 and the reflowed 
line. Additionally, the freefonm digital ink annotation 56 is adjusted to adapt to the one line being removed between the 
anchor points at 66 and 68 to continue to indicate its logical relationship to the text. 

[002S] An embodiment of the method and system of the present invention involves at least a three-step process 
The first step is adding the freeform digital ink annotation on objects in a first view. The second step involves mapping 
segments of the freeform digital ink annotations to corresponding objects in the view. The third step involves moving, 
scaling or otherwise transforming the freeform digital ink annotation to maintain the indication of the logical relationship 
between the freeform digital ink annotation and the corresponding objects as the display changes. 
[0027] Optionally, in an embodiment of the invention the freeform digital ink annotations can be used as navigational 
markers. For example, the user can select an annotation in a view and return to the annotated view that existed when 
the annotation was created. 

[0028] The freeform digital ink annotation of objects in changing views of an embodiment of the invention relies on 
an object model for the view. In the example of Figs. 2A arid 2B, the objects are the nodes in the hyperbolic browser 
In the example of Figs. 3A and 3B, the objects are the text characters. In a third example (not shown), the objects are 
the output from an object tracking system running on a video data stream. 

[0029] An embodiment of the method and system of the invention maps the freeform digital ink annotations to objects 
in a view by placing anchors in the underlying document. For example, a simple circling, highlighting or a crossing out 
mark is anchored tojhe corresponding span of character positions that arecovered by the freeform digital ink annotation 
in the view. In the example shown in Figs. 3A and 3B, a comment 58 in the margin is anchored to a position in the 
document m the margin near all of the corresponding words or characters. 

[0030] More complex freeform digital ink annotations may have several anchors. For example, as shown in Figs. 3A 
and 3B, consider a freelorm digital ink annotation 56 that includes a circle 6B connected to an arrow 66. The circle 68 
is anchored to the span of words or characters that is selected by the circle and the tip of the arrow 66 is anchored to 
its end point while the line connecting the circle 68 and the tip of the arrow 66 is not anchored. 
[0031] An even more complex freeform digital ink annotation may involve logical relationships to moving objects in 
a virtual world. An example of such an annotation involves the freeform digital ink annotation of a door to a room in a 
virtual world. The layout of the door changes in a view of the virtual world as the door is opened or closed. The method 
and system of an embodiment of the invention maintain the indication of the logical relationship between the Ireeform 
digrtal mk annotation and the door as the view of the door changes. Thus, the user sees the freeform digital ink anno- 
tation rotating with the door. 

[0032] For the purposes of the following discussion, the following definitions are provided: A position is a two-dimen- 
sional coordinate in a view. A stroke is a list of positions in a view. An offset is a two-dimensional offset from a position 
in a view. A region is a portion of a two dimensional view that is represented, lor example, by a bounding box or by a 
list of positions. A character is a character in a text document. Character positions between characters in the flow of 
text are mamtained as the text changes. A margin anchor is a character position and a boolean flag for left or right 
margins. A simple anchor is a character, a character position or a margin anchor. A composite anchor is a pair of simple 
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anchors with offsets. An anchor is either a simple anchor or a composite anchor. An anchored stroke is an anchor plus 
a list of offsets. 

[0033] An embodiment ol the method and system of the invention moves the anchor points if the underlying anchored 
objects move. Unanchored segments ol the freeform digital ink will stretch to fit between the anchored points. Some- 
times adjacent anchor points move apart because the underlying objects move apart: For example, if a text phrase 
reflows from a single line across two lines (as shown in Figs. 3A and 3B). In such cases, the connection between the 
lormerly adjacent anchored points can be left out to reduce clutter and any unanchored ink between those points can 
be removed. Thus, as shown in Fig. 3B, the reflowed freeform digital ink annotation ts split and does not show ink 
extending between the two segments, which are on two separate text lines. If anchor objects for a stroke are deleted, 
then the freeform digital ink annotations can be remapped to the nearest remaining objects in the document. If all 
anchor objects for the freeform digital ink annotation are deleted then the annotation can be deleted. 
[0034] The type of the anchors for the freeform digital ink annotation depends upon the shape of the annotation and 
the kind and number of the corresponding objects. One method, detailed below, for computing ink anchors is to select 
the nearest object for each segment in the ink. Alternatively, the type of stroke, as determined from its shape, can be 
used to determine the anchor objects. In another case, a freeform digital ink annotation with multiple anchors may 
cause unwanted distortions when the view changes. To avoid this problem, an embodiment of the present invention 
groups segments of the freeform digital ink annotation together to reduce the number of anchors. 
[0035] As briefly described earlier, an embodiment of the method and system of the present invention uses at least 
three processes. The first process receives freeform digital ink annotations on the view, the second process computes 
a logical mapping between objects in the view and the freeform digital ink annotations, and the third process renders 
the freeform digital ink annotations on a view that has changed. An optional fourth process selects freeform digital ink 
annotations to navigate to a previous state. 

[0036] The first process includes sensing data points from a pointing device, such as a mouse or a pen, rendering 
the ink strokes by connecting the points on the view, and grouping the ink strokes together to form a freeform digital 
ink annotation. These ink strokes may be grouped, tor example, by a simple time threshold. Tho second process 
receives the freeform digital ink annotations as input and maps between the objects and the freeform digital ink anno- 
tations. The third process updates the view when the document changes. This process uses the mapping to render 
the freeform digital ink annotation on the objects as the view changes. 

[0037] The procedures and the embodiments of the invention described herein assume that the objects do not change 
30 while the user is inking. However, the method and system of the invention is not limited to applications in which, the 
objects being annotated do not change. As an example, in the case of video, the video may be hatted while the user 
marks. Alternatively, the video can continue while the user marks and the state of the objects at some point in time 
can be used to interpret the objects and update the mapping between the objects and the freeform digital ink annota- 
tions. 

[0038] The fourth process takes a selection on a view, determines if it corresponds to a freeform digital ink annotation, 
and, if it does, replaces the current view with the view which existed when the freeform digital ink annotation was made. 
This process requires storing an additional mapping from the freeform digital ink annotations to the state of the changing 
view. 

[0039] The present invention recognizes that objects may be built from sub-objects which may be split from each 
7: other. One example is a text phrase that is built out of words. If a phrase has a Ireeform digital ink annotation that 
• circles the phrase and if that phrase reflows across two lines, the invention recognizes this split and may either reflow 
the freeform digital ink annotation by splitting it up into several segments, each of which follows the corresponding 
underlying words as they move, or the system may choose to draw lines between matching segments 61 the circle. 
Alternatively, the Ireeform digital ink annotation could be replicated at a smaller scale on each individual word. 
[0040] A user can also copy a phrase to another paragraph, which could be considered a form of splitting or as an 
entirely new and unrelated object, without the circle annotation. 

[0041] Conversely, objects can merge into larger entries. For example, it an unannotated word between two anno- 
tated words is deleted then the two annotated words 'merge' to form an annotated phrase. The two annotations join 
to form a single annotation of the merged phrase. 

[0042] Objects may also disappear entirely- For example, words in a text editor can be deleted, tn this case, the 
•freeform digital ink annotations associated with the object can disappear also, or they can remain at the location be- 
tween the surrounding objects. 

[0043] Objects can also change. For example, a user in a text editor can select an annotated word and replace it 
with a new word. The new word could be considered a a new version of the original object. The annotation on the 
original word may then be associated with the new object, tn this case, the annotation may alter its shape to conform 
to the new word, tor example, by stretching or shrinking. Alternatively, the annotation could be removed from the display. 
[0044] An embodiment of the method and system of this invention may group ink strokes on text to form a single 
Ireeform digital ink annotation. By way of non-limiting examples, the annotations can be categorized as circles, under- 
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Itnes/cross-outs. insertions, margin notes, and connectors. Annotations that are circles are identified by ink stroke 
circles that contain text and these ink stroke circles are mapped to the text objects inside the ink stroke circle Underlined 
crosscut annotations are .dentified by elongated marks which are either directly on the text or underneath the texL 
These annotations are mapped to the text objects that they are on or underneath. An annotation may be identified as 
an insertion if the annotations dips into a line of text. An insertion annotation maps to the position where the insertion 
mark appears in the text, for example, between two characters. A margin note is an annotation which is located in a 
margin. The margm note annotations are mapped to the text that is adjacent to the vertical and/or horizontal position 
of the annotations. Connector annotations are identified as connectors by arrows that may appear at the end of the 
annotation or by lassos (i.e., a circle plus a line and a single stroke). A connector annotation is split into a circle and a 
connector portion. Connector annotations are mapped to two objects independent*. The first object is the object that 
occurs at the start of an annotation and the second object occurs near the end of the annotation These objects may 
be positions in the text, or they may be other annotations. As an example, one end of the line portion of a lasso is 
mapped to the circle annotation which, in turn, maps to the text. Another example is a line from the text to a margin 
annotation. One end of the line is mapped to the text while the other end is mapped to the margin annotation 
[0045] Fig. 6 shows several exemplary types of treeform digital ink annotations. Fig. 6 shows a display of a document 
400 including circle {or enclosure) annotations 402 and 404, an underline annotation 406, crossout annotations 408 
and4l0, insertion annotations 412 and414, a margin annotation 416, a line to margin annotation 418 (with associated 
margin note 420) and an arrow connector 422. It should be understood that the types of freeform digital ink annotations 
shown m conjunction with the descriptions of various exemplary embodiments herein are intended to be illustrative 
and not limiting to any particular embodiment/type or grammar of annotation. 

[0046] The method and system of this invention is based upon the concept that anchors are determined for freeform 
digital ink annotations. Anchors are offsets from objects. Freeform digital ink annotations are mapped to the object 
structure of the view underlying the freeform digital ink annotations. While it is to be understood that the mapping may 
be to h.erarchical object structures such as characters, phrases and paragraphs, for the sake of simplicity, the following 
description describes a mapping of a single level: between freeform digital ink annotations andcharacters or positions 
between characters. A text editor maintains these characters and positions as the text changes. For characters that 
have been deleted, the text editor marks the characters and their corresponding character positions as deleted but 
the text editor still maintains a correspondence between the character positions and some location between remaining 
characters. 3 

[0047] An embodiment of the method and system of the invention may rely upon a text editor that stores the sequence 
and presentation of characters. The embodiment can determine if a character position, a character or a range of char- 
acters has been deleted; can acquire the position in the view that corresponds to a character position* acquire the 
region in the view that corresponds to a character: can obtain left and right margin positions that correspond to a 
character posttion; can acquire a range of character positions that correspond to a region in a view and can split a 
freeform digital ink annotation into a set of substrokes or segments, such that each substroke corresponds to a single 
character or to a position between characters. 

[0048] One embodiment of the method and system of the invention may afeo use an annotation database that stores 
anchored strokes. The annotation database may be accessed to add an anchored stroke to the database to delete 
an anchored stroke from the database or to acquire a list of all of the anchored strokes. 

[0049] Figs. 4A through 4F show a flowchart outlining a control routine for mapping freeform digital ink annotations 
to objects m an undertying view in accordance with an embcxJimentof the invents 
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First, classify the group of strokes as circles, underlines/cross-outs, insertions, margin 
notes, or connectors. Notice that this may require splitting some of the strokes, ami 
may result in several groups of strokes. Then, 

Sort the new groups of strokes so thai connectors appear afler any groups they 
connect to 

For each new group of Strokes 

If the group is a circle or an underline or a cross-out, then 

Split the group of strokes into a list of strokes and characters 
For each stroke and character pair in the list 

Create a new simple anchor corresponding to the character 

Get the region corresponding to the character 

Let anchor position be the center of the region 

Create a new anchored stroke based on the simple anchor 

For each position in the stroke, 

Let offset be position minus anchor position 
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Append offset to the anchored stroke's list of offsets 

End for 

Add anchored stroke to the annotation database 

End for 

Else if the group is an ink insertion into the text, then 

Let character position be the character position where the ink dips into the 
text 

Get the position corresponding to the character position 
Let anchor position be that position 
For each stroke in the group 

Create a new anchored stroke based on the character position 
For each position in the stroke 

Let offset be position minus anchor position 
Append offset to the anchored stroke's list of offsets 

End for 

Add anchored stroke to the annotation database 
End for 

Else if the group is a margin annotation, then 

Let character position be the first character position corresponding to the 
vertical extent of the annotation 

Get the lea and right margin positions corresponding to character position 
If the group is mostly to the left of the left margin position, then 

Let margin anchor be { character position, flag=left } 

Let anchor position be the left margin position 

Else 

Let margin anchor be { character position, flag=right } 
Let anchor position be the right margin position 

End if 

For each stroke in the group, 

Create a new anchored stroke based on the margin anchor 

For each position in the stroke 

Let offset be position minus anchor position 
Append offset to the anchored stroke's list of offsets 

End for 

Add anchored stroke to the annotation database 
End for 

Else if the group is a connector, then 

For each of starting and ending position on connector 
If position is near an annotation 

Let simple anchor be the anchor for that annotation 
Let anchor position be the current position of the simple 
anchor 

Let offset be position minus anchor position 
Else (position is near an object) 

Let simple anchor be the anchor for that object 

Let anchor position be the current position of the object 

Let offset be position minus anchor position 

End if 

End for 
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Let composite anchor be the anchors and offsets corresponding to the 
starting and ending positions 
5 For each stroke in the group 

Create a new anchored stroke based on the composite anchor 
For each position in the stroke 

Let offset be position minus starting position 
10 Divide offset by the distance between thestarting and 

ending positions 

Rotate offset around the origin by-angle (ending position 
minus starting position) 

is Append oflset to the anchored stroke's list of offsets 

End for 

Add anchored stroke to the annotation database 

End for 

20 End if 

End for 

[0050] The control routine shown in Figs. 4A-4F starts at step S100 and continues to step S102. In step SI 02, the 
control routine groups the ink strokes into annotations such as circles, underlines/cross-outs, insertions, marginalia or 

2S connectors. It is to be appreciated that this may require splitting some strokes and may result in several groups of 
strokes. The control routine then continues to step S104 where the annotations are sorted so that connectors appear 
alter any group to which they are connected. The control routine then continues to step SI 06 where the control routine 
initiatesa loop that iteratively processes all of these annotations. The control routine determines if the current annotation 
is a circle, an underline or a cross-out. If in step S106. the control routine determines that the current annotation is a 

30 circle, an underline or a crosscut then the control routine jumps to step S116. If, on the other hand, in step S106 the 
control routine determines that the current annotation does not include a circle, an underline or a cross-out then the 
control routine continues to step S108. In step S108, the control routine determines if the current annotation is an ink 
insertion. If, in step S1Q8, the control routine determines that the current annotation is an ink insertion then the control 
routine jumps lo step St 38. If, on the olher hand, the control routine determines that the current annotation is not an 

35 ink insertion, then the control routine continues to step StiO. In step Si 10, the control routine determines if the if the 
current annotation is a margin annotation. If, in step S110, the control routine determines that the current annotation 
is a margin annotation then the control routine jumps to step S158. If, on the olher hand, in step S 110. the control 
routine determines that the current annotation is not a margin annotation then the control routine jumps to step S186. 
[0051] After completing one of the processes of Figs. 4B-4F, which will be explained in detail below, the control 

to routine continues to step S112 where the control routine determines if more annotations need to be processed. If in 
step S1 12. the control routine determines that more annotations need to be processed, then the control routine returns 
to step S106. II, on the other hand, the control routine determines that no more annotations need to be processed, 
then the control routine continues to step S1 1 4 where control is returned to the environment in which the control routine 
of Figs. 4A-4F operate. 

[0052] At step Sll 6, the control routine splits the annotation into a list ol substrokes each with a corresponding single 
character. Thus, the control routine splits the annotation into substroke/eharacter pairs. The control routine then con- 
tinues to step Si 18. where the control routine initiates a loop to ileralivery process each substroke/character pair in 
the current annotation. In step S118, the control routine creates a new simple anchor that corresponds to the current 
character. The control routine then continues to step S120 where the control routine acquires the regbn in the view 
that corresponds to the current character. The control routine then continues to step S122 where the control routine 
sets an anchor position to the center of the region in the view. The control routine then continues to step S1 24 where 
the control routine creates a new anchored stroke that is based upon the simple anchor and continues to step S126. 
In step $126, the control routine sets an oflset to the position from the substroke minus the anchor position and con- 
tinues to step S128. In step S128. the control routine appends the oflset to the anchored stroke's list of offsets and 
continues to step Si 30. In step SI 30, the control routine determines whether more positions exists in the current 
substroke. If the control routine determines in step S1 30 that more positions exists in the current substroke, then the 
control routine returns lo step S126. II. on the other hand, the control routine determines that more positions do not 
exist in the current substroke then the control routine continues to step S132. In step SI 32, the control routine adds 
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the anchored stroke to the annotation database; and the control routine then continues to step S134. In step S134, 
the control routine determines if more substroke/character pairs exist in the current annotation. If. in step St 34, the 
control routine determines that more substroke/character pairs exist in the current annotation then the control routine 
returns to step S118. If, on the other hand, the control routine determines, in step S134, that no more sunstroke/ 
character pairs exist in the current annotation then the control routine continues to step SI 36 where control returns to 
stepS112. 

[0053] At step S 1 38 the control routine sets a current character position to the character position where the annotation 
dips into the text. The control routine then continues to step S140 where the control routine acquires the position 
corresponding to the current character position. The control routine then continues to step S142 where an anchor 
position is set to the current character position. The control routine then continues to step S144 where the control 
routine initiates a loop to process atl strokes in the annotation. In step S1 44, the control routine creates a new anchored 
stroke based upon the current character position as an anchor and the control routine continues to step S146. In step 
S 1 46, the control routine sets an offset to the current character position from the stroke minus the anchor position and 
continues to step S148. In step S148, the control routine appends the offset to the anchored stroke's list of offsets and 
continues to step SI 50. In step SI 50, the control routine determines whether more positions exists in the current stroke. 
If, in step SI 50, the control routine determines that more positions exist in the current stroke then the control routine 
returns to step S146. If, on the other hand, in step S150, the control routine determines that no more positions exist 
in the current stroke then the control routine continues to step S1 52. In step SI 52. the control routine adds the anchored 
stroke to the annotation database and the control routine continues to step SI 54. In step SI 54, the control routine 
determines whether more strokes exist in the current annotation. II, in step S154, the control routine determines that 
more strokes exist in the current annotation then the control routine returns to step S144. If, on the oiher hand, the 
control routine determines, in step S154, that no more strokes exists in the current annotation then the control routine 
continues to step S112. 

[0054] At step SI 58, the control routine sets a character position to the first character position corresponding to the 
vertical extent of the current annotation. The control routine then continues to step S160 where the text editor acquires 
the left and right margin positions that correspond to the character position and continues to step S162. In step S162, 
the control routine determines whether the annotation is mostly to the left of the left margin. If, in step S1 62, the control 
routine determines that the annotation is mostly to the left of the left margin, then the control routine continues to step 
SI 64. If, on the other hand, the control routine determines that the annotation is not mostly left of the left margin then 
the control routine jumps to step SI 68. In step S1 64. the control routine sets the margin anchor to LEFT and the control 
routine continues to step S1 66. In step S1 66. the control routine sets an anchor position to the left margin position and 
continues to step S172. 

[0055] In step S168. the control routine sets a margin anchor to RIGHT and continues to step S1 70. In step SI 70, 
the control routine sets an anchor position to the right margin position and continues to step S172. In step S172, the 
control routine initiates a loop to process all strokes in the current annotation. In step S1 72, the control routine creates 
a new anchored stroke based upon the margin anchor and continues to step S174. In step S174, the control routine 
sets an offset to the position from the current stroke minus the anchor position and continues to step S176. In step 
S1 76, the control routine appends the offset to the anchored stroke's list of offsets and continues to step S178. In step 
S178, the control routine determines whether more positions exist in the current stroke. If, in step S178, the control 
routine determines that more positions exist in the current stroke then the control routine returns to steps S174. If, on 
the other hand, in step S178, the control routine determines that no more positions exists in the current stroke then 
the control routine continues to steps S180. In step S 1 80, the control routine adds the anchored stroke to the annotation 
database and continues to step S182. In step S182, the control routine determines whether more strokes exists in the 
current annotation. If, in step S182, the control routine determines that more strokes exists in the current annotation, 
then the control routine returns to step S172. If, on the other hand, the control routine determines, in step S182, that 
no more strokes exists in the current annotation then the control routine continues to step S156 where the control 
routine returns to step S112. 

[0056] At step S186, the control routine initiates a loop to process all starting or ending positions in the current 
annotation. In step S186, the control routine determines whether the current of the starting and ending positions are 
near another annotation or an object. If in step S186, the control routine determines that the current position is near 
another annotation then the controfter continues to step Si 88. If, on the other hand, the control routine determines, in 
step S186, that the position is not near another annotation then the control routine continues to step S192. In step 
S188. the control routine sets a simple anchor as the anchor for the current annotation and continues to step S1 90. In 
step S 190. the control routine sets an anchor position as the current position of the simple anchor and continues to 
step S1 96. In step S1 92, the control routine lets a simple anchor be the anchor for an object which is near the current 
starting or ending position and the control routine continues to step SI 94. In step Si 94. the control routine sets an 
anchor position to the current position of the object and the control routine continues to step Si 96. In step S196. the 
control routine sets an offset to the current position minus the anchor position and continues to step Si 98. In step 
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Si 98. Ihe control routine determines whether more starting or ending positions exist in the current annotation. If, in 
step SI 98. the control routine determines that more starting or ending positions exist in the current annotation then 
the control routine returns to step SI 86. If. on Ih8 other hand, the control routine determines, in step S198. that no 
more starting or ending positions exist in the current annotation then the control routine continues to step S200. 

5 [0057] In step S200. the control routine initiates a loop to process all strokes in the current annotation. In step S20O, 
the control routine creates an anchored stroke having a composite anchor based upon the anchors corresponding to 
the starting and ending positions and continues to step S202. In step S202. the control routine sets an offset to the 
current position in the current stroke minus the starling position and continues to step S204. In step S204, the control 
routine divides the offset by the distance between the starting and ending positions and continues to step S20S. In 

10 step S206. the control routine rotates the offset around an origin by an angle equal to the ending position minus the 
starting position and continues to step S208. In step S208, the control routine appends the offset to the anchored 
stroke's list of offsets and continues to step S210. In step S210, the control routine determines whether more positions 
exists in the current stroke. If. in step S210, the control routine determines that more positions exists in the current 
stroke then the control routine returns to step S202. II. on the other hand, the control routine determines, in step S210, 

*5 that no more positions exists in the current stroke then the control routine continues to step S212. In step S212, the 
control routine adds the composite anchor to the annotation database and continues to step S214. In step S214, the 
control routine determines if more strokes exists in the current annotation. It, in step S21 4, the control routine determines 
that more strokes exists in the current annotation then the control routine returns to step S200. If. on the other hand, 
the control routine determines that no more strokes exists in the current annotation, then the control routine continues 

20 to step S216 where the control routine returns to step St 12. 

[0058] Figs. 5A-5B shows a control routine for rendering freeform digital ink annotations on a changed view. The 
flow chart parallels the following pseudocode. 

2$ Using the annotation database, get the list of all anchored strokes 

For each anchored stroke, 

Extract the anchor and the list of offsets 
Let stroke be a new stroke 
30 If anchor is a simple anchor 

Determine if the character or character position underlying the anchor 
has been deleted 
If not, then 

Let anchor position be the position of that anchor 
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15 



For each offset, 

Let new position be offset + anchor position 
Add new position to the stroke 

End for 

End if 

Else 

Determine if the character or character positions underlying the l a and 
2 0d anchors has been deleted 
If both have been deleted, then 
do nothing 

Else 

Let starting position be the position of the l 11 anchor, plus the 
Toffset 

Let ending position be the position of the 2* anchor, plus the 
2 nd offset 

20 For each offset, 

Let new offset be offset rotated byangle (ending 

position minus starting position) 

Multiply new offset by the distance between the starting 

2B ^d ending positions 

Let new position be new offset plus starting position 
Add new position to the stroke 
End for 

End if 

30 _ j _ 

End if 

Render stroke on the view 

End for 



35 



40 



45 



55 



[0059] The control routine starts at step S300 and continues to step S302. In step S302, the control routine acquires 
a list of all anchored strokes from the annotation database and continues to step S304. In step S304, the control routine 
extracts the anchors and the list o( offsets from the current anchored stroke and continues to step S306. In step S306, 
the control routine lets the stroke be a new stroke and continues to step S308. In step S308. the control routine deter- 
mines whether the anchor is a simple anchor If. in step S308, the control routine determines that the anchor is a simple 
anchor then the control routine continues to step S310. II, on the other hand, the control routine determines, in step 
S308. that the anchor is not a simple anchor then the control routine jumps to step S326. In step S310. the control 
routine determines if the character or character position has been deleted. If, in step S310. the control routine deter- 
mines that the character or character position has been deleted then the control routine jumps to step S320. If, on the 
other hand, the control routine determines that the character or the character position has not been deleted then the 
control routine then continues to step S312. In step S312, the control routine sets me anchor position to the position 
of that anchor for the offset and continues to step S314. In step S314. the control routine lets the new position be the 
offset from the stroke plus the anchor position and continues to step S316. In step S316, the control routine adds the 
new position to the stroke and continues to step S3ia In slep S318, the control routine determines whether there are 
more offsets for the anchor. If, in step S318, the control routine determines that there are more offsets then the control 
routine returns to step S314. It, on the other hand, the control routine determines that there are no more offsets then 
the control routine jumps to step S320. In step S320, the control routine renders the stroke on the view and continues 
tostep S322. In slep S322, thecontrol routine determines if there are more anchored strokes in the annotation database. 
If. in step S322, the control routine determines that there are more anchored strokes in the annotation database then 
the control routine returns to step S304. If, on the other hand, the control routine, determines in step S322, that there 
are no more anchored strokes in the annotation database then the control routine continues to step S324 where the 
control routine returns control to the environment in which the control routine operates. 

[0060] In step S326, the control routine determines if the character or character position underlying the first and 
second anchors been deleted If. in step S326. the control routine determines that both of these character or character 
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positions have been deleted then the control routine returns to step S320. If, on the other hand, the control routine 
determines, in step S326. that the character or character position underlining the first and second anchors have not 
been deleted then the control routine continues to step S328. In step S328. the control routine sets the starling position 
to the position ol the first anchor plus the first offset and continues lo step S330. In step S330. the control routine sets 
the ending position to the position of the second anchor plus the second oflsel and continues lo step S332. In step 
S332, the control routine sets the new offset to the offset from the anchored stroke rotated by an angte equal to ending 
position minus starting position and the control routine continues to step S334. In step $334, the control routine mul- 
tiplies the new offset by the distance between the starling and ending positions and continues to step S336. In step 
S336, the control routine sets the new position to the new offset plus the starting position and continues to step S338. 
In step S338. the control routine adds the new position to the stroke and continues to step S340. In step S340, the 
control routine determines whether more offsets exist. If. in step S340, the control routine determines that more off sets 
exist then the control routine returns to step S332. If, on the other hand, the control routine determines, in step S340, 
that no more offsets exists then the control routine returns to step S320. 

[0061] Freetorm digital ink annotation is intended to mean a visible ink stroke of arbitrary shape made with a pen, a 
mouse or other pointing device that persists over time. A pen based input device is intended to include a pen tablet, 
a pen display, a pen display tablet, a pen computer anoVor a touch screen. The systems and methods ol this invention 
may inter some meaning from the shapes or the morphological characteristics of ink strokes but do not remove the ink 
stroke once they have been recognized. 

[0082J While the detailed description refers generally to electronic systems, it is intended to be understood that the 
method and system of the present invention may be applied lo any electronic book and document reader that displays 
changing views. Electronic book and document readers include any electronic handheld device that can be accessed 
with a pointing device such as a pen and that accepts freeform digital ink as input. Electronic book and document 
readers may also include other additional features without limitation. 

[0063] It is intended to be understood that the term lexf is intended to include text, digital ink, audio, video or any 
other content of a document, including the document's structure. An object is intended to be any component of a display 
such as, by way of non-limiting example, a character, a word, a phrase, a sentence, a paragraph, a line, a section, a 
chapter, a portion ol an image that is identifiable by a computer vision system in a single image or in a stream ol images 
as in a video, etc. 

[0064] It should be understood that a change in the structure of a document is intended to include inserting, deleting, 
copying and moving of objects or the Iflce. It is also intended to be understood that a change in the layout structure of 
a document includes reflowing of objects, resizing of objects, rotation of objects, stretching of objects, distortion of 
objects, morphing of objects or the like. 

[0065] While this invention has been described in conjunction with the specific embodiments outlined above, it is 
evident that many alternatives, modifications and variations are apparent to those skilled in the art. Accordingly, the 
preferred embodiments of the invention as set forth above are intended to be illustrated and not limiting, \fcrious chang- 
es may be made without departing from the spirit and scope ol the invention. 



Claims 

.1 . A method for maintaining an indication of a logical relationship of at least one freeform digital ink annotation to at 
least one object in a first view as the first view changes, the indication including a layout relationship of the at least 
one freeform digital ink annotation to the at least one object, the method comprising: 

mapping the logical relationship ol the at least one freeform digital ink annotation to the at least one object 
based upon the layout relationship ol the at least one freeform digital ink annotation and the at least one object 
in the first view, 
generating a second view; 

rendering the at least one freeform digital ink annotation in response to the generation of the second view, the 
layout relationship of the at least one freeform digital ink annotation and the at toast one object in the second 
view based upon the mapped logical relationship of the at least one freeform digital ink annotation to the at 
least one object in the first view. 

2. The method of claim 1 , wherein the second view is generated in response toat least one ol a change in the structure 
ol a document and a change in the layout of the document. 

3. The method of claims 1 or 2, wherein mapping the logical relationship comprises: 
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splitting the at least one Ireeform digital ink annotation into a plurality of sunstrokes, each substroke corre- 
sponding to a portion of the annotation that is in proximity to an object; and 

creating an anchor for each of the plurality of substrokes based upon the layout relationship of each of the 
plurality of substrokes to at least one object in the first view. 

4. The method of claim 3, wherein: 

rendering the at least one freeform digital ink annotation comprises rendering the at least one f reeform digital 
ink annotation in response to a separation of the pluraBty of objects in the second view; and 
splitting the at least one freeform digital ink annotation further comprises splitting the at least one free form 
digital ink annotation into at least two freeform digital ink annotations, each freeform digital ink annotation 
corresponding to at least one of the plurality of objects. 

5. The method of any of claims 1 -4, wherein: 
the at least one object comprises at least two objects; and 

generating the second view comprises generating the second view in response to at least one of merging at 
least two of the at least two objects and replacing at least one object of the at least two objects with at least 
one new object. 

a The method of any of claims 1 -5. further comprising mapping the logical relationship based on at teas! one of a 
shape of the at least one freeform digital ink annotation, a proximity of the at least one Ireeform digital ink annotation 
to the object, and a location of the at least one freeform digital ink annotation in the first view. 

7. A system for maintaining an indication of a logical relationship of at least one freeform digital ink annotation to at 
least one object in a view as the view changes, the indication including a layout relationship of the at least one 
freeform digital ink annotation to the at least one object, the system comprising: 

a display that displays the layout relationship of the at least one freeform digital ink annotation to the at least 
30 one object; and 

a processor that maps the at least one freeform digital ink annotation to the at least one object based upon 
the layout relationship of the at least one freeform digital ink annotation to the at least one object in a first view 
the display (being] responsive to a change in the first view to generate a second view and to render the at 
least one freeform digital ink annotation in the second view, the layout relationship of the at least one freeform 
digital ink annotation and the at least one object in the second view based on the mapped logical relationship 
of the at least one freeform digital ink annotation to the at least one object in the first view. 

The system of claim 7, wherein the processormaps the at least one freeform digital ink annotation by splitting the 
at least one Ireeform digital ink annotation into a plurality of substrokes, each substroke corresponding to the 
portion of the freeform digital ink annotation that is in proximity to an object, and by creating an anchor for each 
of the plurality of substrokes based upon the layout relationship of each ol the plurality of substrokes to at least 
one object in the first view. 

9. The system of claims 7 or 8, wherein the processor generates the second view in response to at least one of the 
at least one object being split into at least two objects and the at least two objects being separated, the processor 
splitting the at least one freeform digital Ink annotation into at least two freeform digital ink annotations, each 
Ireeform digital ink annotation corresponding to at least one ol the at least two objects. 

10. The system of any of claims 7-9, wherein: 
the at least one object comprises at least two objects; and 

the processor generates the second view in response to at least one of at least two ol the at least two objects 
being merged together and at least one of the at least two objects being replaced with at least one new object. 
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The system of any of claims 7-10, wherein the processor maps the at least one freeform digital ink annotation 
based on at least one of a shape of the at least one freeform digital ink annotation, a proximity of the at least one 
freeform digital ink annotation to the at least one object and a location of the at least one freeform digital ink 
annotation in the first view. 
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The system of any of claims 7-11 . wherein: 

the at (east one freetorm digital ink annotation includes al least two freelorm digital ink annotations and the at 
least one object includes at least two objects, each object corresponding to one of the at least two f reeform 
digital ink annotations: and 

the processor merges at least two of the at least two freeform digital ink annotations into a single Ireelorm 
digital ink annotation in response to at teast two of the at least two objects being merged. 
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402 400 

2 The Problems 

h (his section we present <&mbej)ol problems Ihot stand in the iov of m 
Eoch problem s desaibed withofew concrete examples, end thenT ownuK d 

Qhpl moyTeod to o solution ore presented n subsections. . - — ' 

408. \ 

Z1 Providing odormatnn(occess on mobile devices b hard ( 

V 410 v 404 - 

Specific examples ot this^febfem-^ ' 

r J» Jot being able to get the inlormolkin you need when you're otoy from your desktop. 
^ g • Wailing tor Web pages to download over o slot modem or wireless link. 

One promise of portable computers is information occess anytime, anywhere. People hove 
become reOonl on continuous, hmh-speed, low-cost networks ot their desktop, yet portable 
cwnpulers ore rorefv connecled fo such networks. Codling is o lechmaue that mokes 
41£jwtoble ^computers less dependent on ubiquitous tow-cost nelworkmg. 

416 Coching\m ancient computer sconce tedm^e/C^onTo copy of frequently 

referenced information near ot hand. Although it is a simple concept, o number of subtle 
problems arise in practice. 



iA Coch'^ cdfilionol impfcolion? for mobile devices. When connecting over wireless 
4U i o nelf cochc n ^ ses TOl orf T loke o relo&f^ long line to process, but aba may 
^^^J^esuft in expensive fees for network occess. Moreover, then operoling (feconnecled from o 
' ne ™rt, coche misses moy stop users from continuing with their work. 

^Jj^ ^2.1.1 Removing documents from focal storage 



) 4ie 



Researches hove been investigating new replacement policies thot better predict future 
references. One avenue is to augment computational predictions with direct user 



418 interactions, on the assumption thot people often know which informolion is most 
4 2 0 inportont to keep h bond The TeleWeb [Schi96j and UowgR [Lib>berg95l mobile web 
browsers cBoi users to lock documents into the coche. One severe problem with a user- 
level locking interfere is that over lime more end more documents become locked 
becouse users tend to lock more then they purge. In this cose, caching becomes much less 
r elfedive becouse locked, but no longer relevant, documents con take up o forge port of o 
cache. 
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jpproach is to ask the user which documents to discord instead of which 
its to keep [ColdbergM]. For example, when coche replacement is necessary the 
system might use o 

FIG.6 



BNSOOCtDc <EP IOl8ge3A7 I > 



29 



Date of publication A3: 
27.12.2000 Bulletin 2000/52 



01) EP1 016 983 A3 

EUROPEAN PATENT APPLICATION 

(51) Intel.?: G06F 17/24 




Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 
(88) 

(43) Date of publication A2: 

05.07.2000 Bulletin 2000727 

(21) Application number 99310482.7 

(22) Date ot filing: 23.12.1999 



(84) Designated Contracting States: 

AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 
MC NL PT SE 

Designated Extension States: 
ALLTLVMK RO SI 

(30) Priority: 30.12.1998 US 222962 

(71) Applicant: Xerox Corporation 
Rochester, New York 14644 (US) 



(54) 

(57) The method and system of the invention main- 
lains the indication of the logical relationship of freeform 
digital ink annotations to objects in an electronic display 
even as the view changes. The invention maps the log- 
ical relationship ol the Ireeform digital ink annotations 
to objects in the view based upon the layout relafion- 



(72) Inventors: 

• Schilit, William N 

Menlo Park, California 94025 (US) 

• Price, Morgan N 

Palo Alto, California 94304 (US) 

(74) Representative: Skone James, Robert Edmund 
GILL JENNINGS & EVERY 
Broadgate House 
7 Eldon Street 
London EC2M 7LH (GB) 



ships of the freeform digital ink annotations to the ob- 
jects in the view. If the view changes, the invention main* 
tains the layout relationship of the freeform digital ink 
annotations to the objects in the view based upon the 
mapping of the freeform digital ink annotations to the 
objects. 



Method and system for maintaining freeform ink annotations on changing views 




EP 1016 983 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 99 31 9482 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Cdalton of document with indication, where nppr opi ate, 
ot relevant passages 



Relevant 
to claim 



CLASSIFICATION Of THE 
APPLICATION ffata,7) 



US 5 559 942 A (GOUGH MICHAEL L 
24 September 1996 (1996-09-24) 

* column 2, line 19 - line 39 * 

* column 6, line 3 - line 67 * 



ET AL) 



US 5 621 871 A (JAREMKO HARK ET AL) 
15 April 1997 (1997-84-15) 

* column 8, line 12 - line 54; figure 5 

EP 9 992 384 A (CANON KK) 
17 March 1999 (1999-83-17) 

* paragraph [9108] - paragraph [0112); 
figures 25-27 * 



1,7 



U7 



1,7 



G96F17/24 



TECHNICAL FIELDS 
SEARCHED (1nLCL7) 



G66F 



The present search report has been drawn up for a0 dams 



BERLIN 



6 November 2009 



Deane, E 



CATEGORY OF CITED DOCUMENTS 

X : parfcubrty relevant * t&ktn 

Y : particular^ r*l*r«rt * oorttfaed wfth another 

document ot ttm tame category 
A : tKfanotoob4 background 
O : noo-writtan dbefests* 
P : imermetkfla documett 



T : th*ory or pmcfito undwtybg ttw kwention 
E : •affiar potent documcto, but pubfshvd oo, or 
after fh»B»g<lftt* 



L : document cfctd tor cOwr renem 



a : membw ot 0» %vm patu tonJy. wi»m ui >fttQ 



EP 1016 983 A3 



ANNEX TO THE EUROPEAN SEARCH REPORT 
ON EUROPEAN PATENT APPLICATION NO. 



EP 99 31 9482 



This armei Ksts the paten! fomiry members relating to the patent documents died in the abowe-rnentbrted European search report 
The members a/e as contained m the European Patent Oft ice EOP Be on ^ ' 

The Eurivean Patent OHice b in no way babfe for these particulars which are merety given tor the purpose of information 

06-11-2800 



Patent document 
cited in search report 


Pubfcalion 
date 


Patent 1 unity 
members) 


Publication 
date 


US 5559942 


A 


24-09-1996 


NONE 




US 5621871 


A 


15-04-1997 


US 6133925 A 


17-10-2008 


EP 8902384 


A 


17-03-1999 


JP 11086015 A 


30-83-1999 



w For more details about ihb annox : so* Official Journal ot (h* European Patent Onwe, No. 12/82 



3 



BMSrmrv <fp iai«mui i ^ 



